System and method for registration and coordinated manipulation of augmented reality image components

ABSTRACT

A system and method for registration and coordinated manipulation of augmented reality image components includes registering a model of patient anatomy to a first image of the patient anatomy captured using an imaging device to determine a baseline relationship between the model and the first image, tracking movement of a computer-assisted device used to manipulate the imaging device, updating the baseline relationship based on the tracked movement, and generating a composite image by overlaying the model on a second image of the patient anatomy according to the updated relationship. In some embodiments, the model is semi-transparent. In some embodiments, registering the model to the first image includes adjusting the model relative to the first image based on one or more inputs received from a user and generating a model to image transformation based on the adjustments to the model. The model to image transformation captures the baseline relationship.

RELATED APPLICATIONS

This patent application claims priority to and benefit of the filingdate of U.S. Provisional Patent Application No. 62/443,460, entitled“Registration and Coordinated Manipulation of Augmented Reality ImageComponents,” filed Jan. 6, 2017, and to U.S. Provisional PatentApplication No. 62/538,425, entitled “System and Method for Registrationand Coordinated Manipulation of Augmented Reality Image Components,”filed Jul. 28, 2017, which are hereby incorporated by reference in theirentirety.

TECHNICAL FIELD

Inventive aspects are directed towards augmented reality technologies.In one embodiment, an imaging system is provided for use during theperformance of minimally-invasive surgical procedures.

BACKGROUND

The rise of minimally invasive surgery beginning in the 20th century hasafforded patients with certain ailments a less traumatic surgicaltreatment option. One type of minimally invasive surgery, laparoscopy,involves the use of an endoscope (type of imaging device) to image theinternals of a patient. Treatment is provided using surgical instrumentsprovided at the end of long, thin shafts. Minimally-invasive surgery wasrevolutionized with the advent of computer-assisted surgical systems,such as the da Vinci Surgical System commercialized by IntuitiveSurgical. One innovation of the da Vinci Surgical System is theinclusion of a stereoscopic endoscope. When a user views the resultingstereoscopic image of patient anatomy, the user perceives a 3D scene.The view, however, is limited to the surfaces of the patient anatomyvisible to the endoscope and typically does not give the surgeon orother medical personnel a complete view of the relevant patient anatomyas sub-surface features are hidden. Information on the sub-surfacefeatures is available through separately obtained images obtained eitherpre-operatively or inter-operatively using other imaging modalities,such as computed tomography (CT), magnetic resonance imaging (MRI),fluoroscopy, thermography, ultrasound, optical coherence tomography(OCT), thermal imaging, impedance imaging, laser imaging, nanotube X-rayimaging, and/or the like. Unfortunately, it not always easy to determinean alignment or registration between the endoscope images and theseseparately obtained images.

Accordingly, it would be advantageous to have systems and methods forcombining the endoscope and other images into a composite view thatcould simultaneously show both the surface and sub-surface features ofthe patient anatomy. One such approach is to use augmented realityimaging technologies that create composite images by combining endoscopeimages with patient-specific models of the patient anatomy determinedusing the other imaging modalities. This provides the surgeon or othermedical personnel the capability to visualize the anatomical structureof tissue that lies beneath the tissue surface captured in the endoscopeimage.

SUMMARY

The following summary introduces certain aspects of the inventivesubject matter in order to provide a basic understanding. This summaryis not an extensive overview of the inventive subject matter, and it isnot intended to identify key or critical elements or to delineate thescope of the inventive subject matter. Although this summary containsinformation that is relevant to various aspects and embodiments of theinventive subject matter, its sole purpose is to present some aspectsand embodiments in a general form as a prelude to the more detaileddescription below.

In one embodiment, an augmented reality imaging system is integratedwith a computer-assisted surgical system, a da Vinci Surgical System.The da Vinci Surgical System can include one or more roboticmanipulators. An endoscope or a therapeutic surgical instrument can bemounted on each of the one or more robotic manipulator. The one or morerobotic manipulators are used to position or orient an endoscope or atherapeutic surgical instrument mounted thereon. The roboticmanipulators move in response to movements by a user of one or moremaster manipulators. In this way, the user can control the position ororientation of the endoscope or therapeutic surgical instrument mountedon any given robotic manipulator.

In one embodiment, an augmented reality imaging system is integratedwith a computer-assisted surgical system. A preoperative orintraoperative tomographic scan is taken of a given patient. A computermodel is generated from this tomographic scan. This patient thenundergoes a surgical procedure using a computer-assisted surgicalsystem, such as a da Vinci Surgical System. During this surgicalprocedure, an endoscope is used to provide the surgeon performing theoperation with real time images of a surgical site. These real timeimages can he augmented with a computer model of patient anatomygenerated from the tomographic scan. For example, a composite image canbe created in which the computer model is overlaid on top of theendoscope image. The computer model can provide the surgeon withadditional information beyond what can be visibly seen from theendoscope image. For example, the computer model can provide informationof certain subsurface anatomical structures (e.g., vessels, ducts,calyces, bones, tumors, etc.).

A useful composite image includes the computer model component of thecomposite image and the endoscope image component of the composite imagebeing correctly registered, or aligned. During a typical surgicalprocedure, the endoscope captures, within its field of view, images ofcertain patient anatomy. When the endoscope position is moved, thepatient anatomy captured within the endoscopes field of view changes.For the composite image to remain accurate, the computer model componentof the composite image component is adjusted accordingly to account forthe endoscope movement. Several inventive aspects disclosed in theinstant patent application provide an approach for accomplishing thischallenge.

In one aspect, a computer-assisted surgical system including anendoscope is provided. At the start of a surgical procedure, theendoscope is at an initial position and captures within its field ofview an initial endoscope image of a surgical site. A user of thesurgical system is presented with a computer model of the anatomy, thecomputer model including the surgical site and nearby anatomy. Thecomputer model of the anatomy can be overlaid on top of the endoscopeimage to create the composite image. A user input device, e.g., a tablettouchscreen interface, is provided to allow the user to manipulate(e.g., zoom in/out, rotate, etc.) the computer model component of thecomposite image, leaving undisturbed the endoscope image component ofthe composite image. When the user is satisfied with the alignment ofthe computer model component and the endoscope image component, the userlocks the alignment of the two components.

Once initial alignment of the computer model component and the endoscopeimage component is completed, the scaling of the computer modelcomponent of the composite image has been adjusted to correspond to thatof the endoscope image and the coordinate system of the computer modelhas been aligned with the coordinate system of the endoscope image. Inone instance, viewing a composite image is desired. A user commands amovement of the endoscope, causing a change in the endoscope imagecomponent of the composite image. The commanded movement involves thesynchronized movement of various joints of the robotic manipulator onwhich the endoscope is mounted. The amount that each joint is moved is aform of kinematic data that describes the change in position of theendoscope, and accordingly, the change to the endoscope image capturedby the endoscope. In the case in which initial alignment of the computermodel component and the endoscope image component of the composite imagehas already been completed, kinematic data associated with a change tothe endoscope image component of a composite image can be used toproduce a matching, coordinated movement of the computer model componentof the composite image. This method, termed kinematic tracking,preserves the alignment between the computer model component and theendoscope image component when the endoscope is moved during a surgicalprocedure. A chain of transformations used to perform this kinematictracking are detailed below.

Consistent with some embodiments, a method of generating augmentedreality images includes registering a model of patient anatomy to afirst image of the patient anatomy captured using an imaging device todetermine a baseline relationship between the model and the first image,tracking movement of a computer-assisted device used to manipulate theimaging device, updating the baseline relationship based on the trackedmovement, and generating a composite image by overlaying the model on asecond image of the patient anatomy according to the updatedrelationship.

Consistent with some embodiments, a computer-assisted medical deviceincludes an articulated arm, an imaging device mounted on thearticulated arm, and a processor coupled to the articulated arm and theimaging device. The processor is configured to register a model ofpatient anatomy to a first image of the patient anatomy captured usingthe imaging device to determine a baseline relationship between themodel and image, track movement of the imaging device, update thebaseline relationship based on the tracked movement, and generate acomposite image by overlaying the model on a second image of the patientanatomy according to the updated relationship.

Consistent with some embodiments, a non-transitory computer-readablemedium having stored thereon a plurality of machine-readableinstructions which when executed by one or more processors associatedwith a computer-assisted device are adapted to cause the one or moreprocessors to perform a method. The method includes registering a modelof patient anatomy to a first image of the patient anatomy capturedusing an imaging device to determine a baseline relationship between themodel and the first image, tracking movement of a computer-assisteddevice used to manipulate the imaging device, updating the baselinerelationship based on the tracked movement, and generating a compositeimage by overlaying the model on a second image of the patient anatomyaccording to the updated relationship.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a simplified diagram of a computer-assisted system accordingto sonic embodiments.

FIG. 2 is a simplified diagram of a kinematic model of acomputer-assisted medical system according to some embodiments.

FIGS. 3A and 3B are simplified diagrams of displayable images generatedaccording to some embodiments.

FIG. 4 is a simplified diagram of a data flow model for generatingoverlay images according to some embodiments.

FIG. 5 is a simplified diagram of the method of overlaying imagesaccording to sonic embodiments.

Embodiments of the present disclosure and their advantages are bestunderstood by referring to the detailed description that follows. Itshould be appreciated that like reference numerals are used to identifylike elements illustrated in one or more of the figures, whereinshowings therein are for purposes of illustrating embodiments of thepresent disclosure and not for purposes of limiting the same.

DETAILED DESCRIPTION

This description and the accompanying drawings that illustrate inventiveaspects, embodiments, implementations, or applications should not betaken as limiting—the claims define the protected invention. Variousmechanical, compositional, structural, electrical, and operationalchanges may be made without departing from the spirit and scope of thisdescription and the claims. In some instances, well-known circuits,structures, or techniques have not been shown or described in detail inorder not to obscure the invention. Like numbers in two or more figuresrepresent the same or similar elements.

In this description, specific details are set forth describing someembodiments consistent with the present disclosure. Numerous specificdetails are set forth in order to provide a thorough understanding ofthe embodiments. It will be apparent, however, to one skilled in the artthat sonic embodiments may be practiced without some or all of thesespecific details. The specific embodiments disclosed herein are meant tobe illustrative but not limiting. One skilled in the art may realizeother elements that, although not specifically described here, arewithin the scope and the spirit of this disclosure. In addition, toavoid unnecessary repetition, one or more features shown and describedin association with one embodiment may be incorporated into otherembodiments unless specifically described otherwise or if the one ormore features would make an embodiment non-functional.

Further, this description's terminology is not intended to limit theinvention. For example, spatially relative terms-such as “beneath”,“below”, “lower”, “above”, “upper”, “proximal”, “distal”, and thelike-may be used to describe one element's or feature's relationship toanother element or feature as illustrated in the figures. Thesespatially relative terms are intended to encompass different positions(i.e., locations and orientations (i.e., rotational placements) of adevice in use or operation in addition to the position and orientationshown in the figures. For example, if a device in the figures is turnedover, elements described as “below” or “beneath” other elements orfeatures would then be “above” or “over” the other elements or features.Thus, the exemplary term “below” can encompass both positions andorientations of above and below. A device may be otherwise oriented(rotated 90 degrees or at other orientations) and the spatially relativedescriptors used herein interpreted accordingly. Likewise, descriptionsof movement along and around various axes include various special devicepositions and orientations. In addition, the singular forms “a”, “an”,and “the” are intended to include the plural forms as well, unless thecontext indicates otherwise. And, the terms “comprises”, “comprising”,“includes”, and the like specify the presence of stated features, steps,operations, elements, and/or components but do not preclude the presenceor addition of one or more other features, steps, operations, elements,components, and/or groups. Components described as coupled may beelectrically or mechanically directly coupled, or they may be indirectlycoupled via one or more intermediate components.

Elements described in detail with reference to one embodiment,implementation, or application may, whenever practical, be included inother embodiments, implementations, or applications in which they arenot specifically shown or described. For example, if an element isdescribed in detail with reference to one embodiment and is notdescribed with reference to a second embodiment, the element maynevertheless be claimed as included in the second embodiment. Thus, toavoid unnecessary repetition in the following description, one or moreelements shown and described in association with one embodiment,implementation, or application may be incorporated into otherembodiments, implementations, or aspects unless specifically describedotherwise, unless the one or more elements would make an embodiment orimplementation non-functional, or unless two or more of the elementsprovide conflicting functions.

The term “flexible” in association with a part, such as a mechanicalstructure, component, or component assembly, should be broadlyconstrued. In essence, the term means the part can be repeatedly bentand restored to an original shape without harm to the part. Many “rigid”objects have a slight inherent resilient “bendiness” due to materialproperties, although such objects are not considered “flexible” as theterm is used herein. A flexible part may have infinite degrees offreedom (DOE's). Examples of such parts include closed, bendable tubes(made from, e.g., NITINOL, polymer, soft rubber, and the like), helicalcoil springs, etc. that can be bent into various simple or compoundcurves, often without significant cross-sectional deformation. Otherflexible parts may approximate such an infinite-DOF part by using aseries of closely spaced components that are similar to a snake-likearrangement of serial “vertebrae”. In such a vertebral arrangement, eachcomponent is a short link in a kinematic chain, and movable mechanicalconstraints (e.g., pin hinge, cup and ball, live hinge, and the like)between each link may allow one (e.g., pitch) or two (e.g., pitch andyaw) DOF's of relative movement between the links. A short, flexiblepart may serve as, and be modeled as, a single mechanical constraint(joint) that provides one or more DOF's between two links in a kinematicchain, even though the flexible part itself may be a kinematic chainmade of several coupled links. Knowledgeable persons will understandthat a part's flexibility may be expressed in terms of its stiffness.

Unless otherwise stated in this description, a flexible part, such as amechanical structure, component, or component assembly, may be eitheractively or passively flexible. An actively flexible part may be bent byusing forces inherently associated with the part itself. For example,one or more tendons may be routed lengthwise along the part and offsetfrom the part's longitudinal axis, so that tension on the one or moretendons causes the part or a portion of the part to bend. Other ways ofactively bending an actively flexible part include, without limitation,the use of pneumatic or hydraulic power, gears, electroactive polymer(more generally, “artificial muscle”), and the like. A passivelyflexible part is bent by using a force external to the part (e.g., anapplied mechanical or electromagnetic force). A passively flexible partmay remain in its bent shape until bent again, or it may have aninherent characteristic that tends to restore the part to an originalshape. An example of a passively flexible part with inherent stiffnessis a plastic rod or a resilient rubber tube. An actively flexible part,when not actuated by its inherently associated forces, may be passivelyflexible. A single part may be made of one or more actively andpassively flexible parts in series.

In some instances well known methods, procedures, components, andcircuits have not been described in detail so as not to unnecessarilyobscure aspects of the embodiments.

This disclosure describes various instruments and portions ofinstruments in terms of their state in three-dimensional space. As usedherein, the term “position” refers to the location of an object or aportion of an object in a three-dimensional space (e.g., three degreesof translational freedom along Cartesian x-, y-, and z-coordinates). Asused herein, the term “orientation” refers to the rotational placementof an object or a portion of an object (three degrees of rotationalfreedom—e.g., roll, pitch, and yaw). As used herein, the term “pose”refers to the position of an object or a portion of an object in atleast one degree of translational freedom and to the orientation of thatobject or portion of the object in at least one degree of rotationalfreedom (up to six total degrees of freedom). As used herein, the term“shape” refers to a set of poses, positions, or orientations measuredalong an object.

Aspects of the invention are described primarily in terms of animplementation using a da Vinci® Surgical System (specifically, a ModelIS4000, marketed as the da Vinci® Xi™ Surgical System), commercializedby Intuitive Surgical, Inc. of Sunnyvale, Calif., Knowledgeable personswill understand, however, that inventive aspects disclosed herein may beembodied and implemented in various ways, including robotic and, ifapplicable, non-robotic embodiments and implementations. Implementationson da Vinci® Surgical Systems e.g., the Model IS41000; the Model IS4200,commercialized as the da Vinci® X™ Surgical System) are merely exemplaryand are not to be considered as limiting the scope of the inventiveaspects disclosed herein. For example, any reference to surgicalinstruments and surgical methods is non-limiting as the instruments andmethods described herein may be used for animals, human cadavers, animalcadavers, portions of human or animal anatomy, non-surgical diagnosis,industrial systems, and general robotic or teleoperational systems.

FIG. 1 is a simplified diagram of a computer-assisted system 100according to some embodiments. As shown in FIG. 1, computer-assistedsystem 100 includes a computer-assisted device 110 with one or moremovable or articulated arms 120. Each of the one or more articulatedarms 120 may support one or more end effectors. In some examples,computer-assisted device 110 may be consistent with a computer-assistedsurgical device. The one or more articulated arms 120 may each providesupport for one or more tools, surgical instruments, imaging devices,and/or the like mounted to a distal end of at least one of thearticulated arms 120. Computer-assisted device 110 may further becoupled to an operator workstation (not shown), which may include one ormore master controls for operating computer-assisted device 110, the oneor more articulated arms 120, and/or the end effectors. In someembodiments, computer-assisted device 110 and the operator workstationmay correspond to a da Vinci® Surgical System commercialized byintuitive Surgical, Inc. of Sunnyvale, Calif. In some embodiments,computer-assisted surgical devices with other configurations, fewer ormore articulated arms, and/or the like may be used withcomputer-assisted system 100.

Computer-assisted device 110 is coupled to a control unit 130 via aninterface. The interface may include one or more wireless links, cables,connectors, and/or buses and may further include one or more networkswith one or more network switching and/or routing devices. Control unit130 includes a processor 140 coupled to memory 150. Operation of controlunit 130 is controlled by processor 140. And although control unit 130is shown with only one processor 140, it is understood that processor140 may be representative of one or more central processing units,multi-core processors, microprocessors, microcontrollers, digital signalprocessors, field programmable gate arrays (FPGAs), application specificintegrated circuits (ASICs), and/or the like in control unit 130.Control unit 130 may be implemented as a stand-alone subsystem and/orboard added to a computing device or as a virtual machine. In someembodiments, control unit may be included as part of the operatorworkstation and/or operated separately from, but in coordination withthe operator workstation.

Memory 150 may be used to store software executed by control unit 130and/or one or more data structures used during operation of control unit130. Memory 150 may include one or more types of machine readable media.Some common forms of machine readable media may include floppy disk,flexible disk, hard disk, magnetic tape, any other magnetic medium,CD-ROM, any other optical medium, RAM, PROM, EPROM, FLASH-EPROM, anyother memory chip or cartridge, and/or any other medium from which aprocessor or computer is adapted to read.

As shown, memory 150 includes an imaging application 160 that may beused to support the imaging techniques described further below that aidan operator, such as a surgeon or other medical personnel in theoperation of computer-assisted device 110. Imaging application 160 mayinclude one or more application programming interfaces (APIs) forreceiving position, motion, and/or other sensor information fromcomputer-assisted device 110 and/or a surgical table, receiving imagesand/or models from external sources (not shown), interacting with userinterface devices, generating images for display, and/or the like. Andalthough imaging application 160 is depicted as a software application,imaging application 160 may he implemented using hardware, software,and/or a combination of hardware and software.

In some embodiments, computer-assisted system 100 may be found in anoperating room and/or an interventional suite. And althoughcomputer-assisted system 100 includes only one computer-assisted device110 with two articulated arms 120, one of ordinary skill wouldunderstand that computer-assisted system 100 may include any number ofdevices with articulated arms and/or end effectors of similar and/ordifferent design from computer-assisted device 110. In some examples,each of the devices may include fewer or more articulated arms and/orend effectors. In some examples, computer-assisted device 110 may beconsistent with a da Vinci Surgical System.

Computer-assisted system 100 further includes a surgical table 170. Likethe one or more articulated arms 120, surgical table 170 may supportarticulated movement of a table top 180 relative to a base of surgicaltable 170. In some examples, the articulated movement of table top 180may include support for changing a height, a tilt, a slide, aTrendelenburg orientation, and/or the like of table top 180. Althoughnot shown, surgical table 170 may include one or more control inputs,such as a control pendant for controlling the position and/ororientation of table top 180. In some embodiments, surgical table 170may correspond to one or more of the operating tables commercialized byTrumpf Medical Systems GmbH of Germany.

Surgical table 170 may also be coupled to control unit 130 via acorresponding interface. The interface may include one or more wirelesslinks, cables, connectors, and/or buses and may further include one ormore networks with one or more network switching and/or routing devices.In some embodiments, surgical table 170 may be coupled to a differentcontrol unit than control unit 130. In some examples, imagingapplication 160 may include one or more application programminginterfaces (APIs) for receiving position, motion, and/or other sensorinformation associated with surgical table 170 and/or table top 180. Insome examples, imaging application 160 may help registercomputer-assisted device 110 with surgical table 170 so that a geometricrelationship between computer-assisted device 110 and surgical table 170is known. In some examples, the geometric relationship may include atranslation and/or one or more rotations between coordinate systemsmaintained for computer-assisted device 110 and surgical table 170.

Computer-assisted system 100 further includes a user interface (UI)device 190. device 190 may also be coupled to control unit 130 via acorresponding interface. The interface may include one or more wirelesslinks, cables, connectors, and/or buses and may further include one ormore networks with one or more network switching and/or routing devices.UI device 190 further includes one or more controls and/or other inputmechanisms for receiving input and commands from an operator. In someexamples, UI device 190 may include a touch-sensitive input mechanism,such as a touch screen, a tablet, a digitizer, a telestrator, and/or thelike for receiving the inputs and commands from the operator. UI device190 further includes one or more display units or screens for displayingvarious images to the operator as is described in further detail below.In some examples, UI device 190 may include left and right outputdisplays to provide 3D stereoscopic images to an operator. In someexamples, the touch-sensitive input mechanism may be combined with orseparate from the one or more display screens.

FIG. 2 is a simplified diagram of a kinematic model 200 of acomputer-assisted medical system according to some embodiments. As shownin FIG. 2, kinematic model 200 includes kinematic information associatedwith many sources and/or devices. The kinematic information may be basedon known kinematic models for the links and joints of acomputer-assisted medical device and a surgical table, such ascomputer-assisted device 110 and surgical table 170, respectively. Thekinematic information may be further based on information associatedwith the position and/or orientation of the joints of thecomputer-assisted medical device and the surgical table. In someexamples, the information associated with the position and/ororientation of the joints may be derived from one or more sensors, suchas encoders and resolvers, measuring the linear positions of prismaticjoints and the rotational positions of revolute joints. In someexamples, the information associated with relative position and/ororientation of the joints may be derived from external trackers, such asoptical trackers, electromagnetic trackers, shape sensors, and/or thelike. In some examples, the kinematic information is not known directlyand is inferred and/or computed based on the known kinematicinformation.

Kinematic model 200 includes several coordinate frames or coordinatesystems and transformations, such as homogeneous transformations, fortransforming positions and/or orientations from one of the coordinatesystems to another of the coordinate systems. In some examples,kinematic model 200 is used to permit the forward and/or reverse mappingof positions and/or orientations in one of the coordinate systems in anyother of the coordinate systems by composing the forward and/orreverse/inverse transformations noted by the transform linkages includedin FIG. 2. In sonic examples, when the transformations are modeled ashomogenous transformations in matrix form, the composing may beaccomplished using matrix multiplication. In some embodiments, kinematicmodel 200 may be used to model the kinematic relationships ofcomputer-assisted device 110 and surgical table 170 of FIG. 1.

As shown in FIG. 2, kinematic model 200 includes a world coordinatesystem (WCS) 210 acting as a reference coordinate system associated withthe operating room, a patient side cart (PSC) coordinate system 220corresponding to a base coordinate system for a computer-assisted device(such as computer-assisted device 110), a gantry coordinate system 230corresponding to portion of the computer-assisted device to which one ormore manipulators may be mounted, an endoscope camera coordinate system240 corresponding to a position and orientation of an endoscope, anendoscope image coordinate system 250 corresponding to images obtainedusing the endoscope, a model coordinate system 260 corresponding to ananatomical model of patient anatomy, a patient coordinate system 270corresponding to the position and orientation of the patient, and atable coordinate system 280 corresponding to the position andorientation of the surgical table (such as table 170). In some examples,the anatomical model is derived from pre-operative or other images ofthe patient anatomy. Kinematic model 200 further includes a userinterface device system 290 corresponding to a coordinate system used tointerpret inputs from the operator used to manipulate the model to helpposition and orient model coordinate system 260.

Kinematic model 200 further includes several transformations used tomodel the relationships between the various coordinate systems. Each ofthe transformations are then used to model various forward and reversekinematic chains that are usable to support the imaging operations asdescribed further below. Each of the transformations shown in FIG. 2denotes a “link” in one of the kinematic chains in kinematic model 200as shown by a corresponding arrow in the figure. Each of thetransformations is identified by the coordinate systems it provides alink between, where the A coordinate system to B coordinatetransformation (or A-to-B transformation) is represented by ^(B)T_(A).

Kinematic model 200 further includes two closed kinematic chains wherethere is more than one set of transformations that may be used to movebetween the same two coordinate systems. One closed kinematic chain isbetween PSC coordinate system 220 and model coordinate system 260. Thisclosed kinematic chain means that the coordinate transformations frommodel coordinate system 260 through endoscope image coordinate system250, endoscope camera coordinate system 240, and gantry coordinatesystem 230 to PSC coordinate system 220 are equivalent to the directtransformation between model coordinate system 260 and PSC coordinatesystem 220 using model-to-PSC transformation 295. This equivalence isshown in Equation 1. This means that model-to-PSC transformation 295 maybe determined from model-to-image transformation 255, image-to-cameratransformation 245, camera-to-gantry transformation 235, andgantry-to-PSC transformation 225.

^(PSC)T_(Model)=^(PSC)T_(Gantry)·^(Gantry)T_(Cameral ·)^(Camera)T_(image)·^(Image)T_(Model)  Equation 1

Another closed kinematic chain includes the loop between worldcoordinate system 210 and model coordinate system 260. The equivalenceof this closed kinematic chain is shown in Equation 2.

^(WCS)T_(PSC)·^(PSC)T_(Model)=^(WCS)T_(Table)·^(Table)T_(Patient)·^(Patient)T^(Model)  Equation2

According to some embodiments, some observations regarding thetransformations are possible. Table-to-WCS transformation 285 istypically not known as the surgical table may typically be moved aboutand oriented flexibly within the operating room. PSC-to-WCStransformation 215 is typically not known as the computer-assisteddevice may similarly be moved about and oriented flexibly within theoperating room. Patient-to-table transformation 275 is typically notknown as the position and the orientation of the patient relative to thesurgical table may vary by patient, procedure, surgeon, and/or the like.However, under the assumption that model-to-patient transformation 265is an identity transformation (i.e., model coordinate system 260 matchespatient coordinate system 270) it is possible to use a registrationprocedure between the computer-assisted device and the surgical table todetermine a table-to-PSC transformation (not shown) that creates aclosed kinematic chain that allows for world coordinate system 210 to bebypassed and patient-to-table transformation 275 to be computed based onthis closed kinematic chain. Several procedures are available forregistering the surgical table to the computer-assisted device. Examplesof these procedures are described in further detail in commonly ownedU.S. patent application Ser. No. 15/522,180, filed on Apr. 26, 2017 andentitled “System and Method for Registering to a Surgical Table,” whichis hereby incorporated by reference in its entirety.

Gantry-to-PSC transformation 225 and camera-to-gantry transformation 235are determined from the kinematics and joint sensor signals that areknown for the computer-assisted device. Image-to-camera transformation245 is determined through one or more calibration procedures associatedwith the endoscope that takes into account the focal length of theendoscope. When the focal length of the endoscope is variable, the oneor more calibration procedures may be used to provide correspondingimage-to-camera transformations 245 that account for the changes infocal length.

Kinematic model 200 is used in two different phases to support theimaging techniques as described in further detail below. In a firstphase, a registration process is used to determine an initial orregistered version of model-to-image transformation 255. Theregistration process includes overlaying the model of the patientanatomy over live endoscope images taken of the same patient anatomy.The operator then manipulates the model using one or more inputmechanisms, such as the input mechanisms of UI device 190. When theoperator is satisfied that the model is correctly positioned and alignedwith the actual anatomy as shown in the overlay, a baseline value formodel-to-PSC transformation 295 is determined by capturing thekinematics of the computer-assisted device to determine gantry-to-PSCtransformation 225 and camera-to-gantry transformation 235, using thecamera calibration information for the endoscope to determineimage-to-camera transformation 245, and applying Equation 1. Thiseffectively registers model coordinate system 260 and patient coordinatesystem 270 so that model-to-patient transformation 265 is an identitytransformation. In some examples, little or no movement of thecomputer-assisted device is typically allowed during the registrationprocess as it can complicate or prolong the registration process,however, the registration may be completed even when gantry-to-PSCtransformation 225, camera-to-gantry transformation 235, andimage-to-camera transformation 245 are not static as long as at theinstant of registration, Equation 1 can be applied to determine thebaseline model-to-PSC transformation 295.

In the second phase, the baseline model-to-PSC transformation 295 isused to adjust model-to-image transformation 255 as the operator drivesthe computer-assisted device and/or adjusts the endoscope (i.e., asgantry-to-PSC transformation 225, camera-to-gantry transformation 235,and image-to-camera transformation 245 change). This is often referredto as tracking. Under the assumption that the surgical table is notmoving and the patient is not moving relative to the surgical table, themodel-to-PSC transformation 295 is static and does not change. Thus,Equation 1 can be rearranged to describe new updated values formodel-to-image transformation 255 that are used to keep the modelmatched to the endoscope images. This is shown in Equation 3.

^(Image)T_(Model)=[^(PSC)T_(Gantry)·^(Gantry)T_(Camera)·^(Camera)T_(Image)]⁻¹·^(PSC)T_(Model)  Equation3

When the surgical table moves or the patient moves relative to thesurgical table, model-to-PSC transformation 295 changes and has to beupdated. In some examples, model-to-PSC transformation 295 may beupdated by repeating the registration process of the first phase. Insome examples, the registration between the surgical table and thecomputer-assisted device and the table-to-PSC transformation thatresults may be used to update model-to-PSC transformation 295. Theupdated model-to-PSC transformation 295 may then be used with Equation 3to determine the updated model-to-image transformation 255.

FIGS. 3A and 3B are simplified diagrams of displayable images generatedaccording to some embodiments. FIG. 3A includes a simplified renditionof an endoscope image 300 consistent with a partial nephrectomyprocedure on a porcine. Visible within endoscope image 300 are a kidney310 and nearby intestines 320. FIG. 3B includes a simplified renditionof a composite image 350 showing an overlay of a model of thecorresponding anatomy over endoscope image 300 after registration of themodel has occurred. In some examples, the model may be generated from CTangiographic imaging of the anatomy. Still visible within compositeimage 350 are kidney 310 and intestines 320. Also included withincomposite image 350 are semi-transparent elements from the modelincluding the parenchyma 360 of kidney 310 as well as non-visibleelements including internal structures 370 of kidney 310, ureter 380,and vasculature 390. As shown, each of the different elements from themodel (parenchyma 360, internal structures 370, ureter 380, andvasculature 390) is shown using different patterns to help the operatordistinguish between different structures, in some examples, other visualdifferentiating techniques may also be used, such as false coloring. Insome examples, semi-transparency of the model elements may be obtainedusing techniques such as alpha blending and/or the like. Overlay of themodel elements on endoscope image 300 as shown in composite image 350allows the operator to simultaneously visualize the interior and hiddenstructures of the anatomy from the model in correct relationship withvisible anatomy from endoscope image 300. This allows, for example, theoperator to avoid areas of the anatomy during a procedure, to identifyhidden features (e.g., a lesion), and/or the like. And although only oneendoscope image 300 and composite image 350 are shown in FIG. 3, whenstereoscopic imaging is used, both left and right versions of theseimages would be generated of which endoscope image 300 and compositeimage 350 are representative examples.

As discussed above and further emphasized here, FIGS. 3A and 3B aremerely examples which should not unduly limit the scope of the claims.One of ordinary skill in the art would recognize many variations,alternatives, and modifications. In some embodiments, other arrangementsand configurations of endoscope image 300 and/or composite image 350 arepossible. In some examples, composite image 350 may include the modelrendered as a picture within picture and/or inset picture within theendoscope image. In some examples, composite image 350 may include themodel rendered above, below, to the right of, and/or to the left of theendoscope image. In some examples, endoscope image 300 may be obtainedusing fluorescence imaging and/or multi-spectral imaging and may includeimages of one or more subsurface anatomical structures, such as lesions,tumors, vasculature, ducts, calyces, and/or the like.

FIG. 4 is a simplified diagram of a data flow model 400 for generatingoverlay images according to some embodiments. In some embodiments, dataflow model 400 may be implemented using control unit 130, imagingapplication 160, and UI device 190 of FIG. 1. As shown in FIG. 4, dataflow model 400 includes a registration module 410, a rendering module420, an overlay module 430, an optional compression module 440, aninterface module 450, and a UI device 460.

Registration module 410 receives kinematic data (e.g., endoscope,computer-assisted device, and table coordinates and sensor values)and/or calibration data for the endoscope. Registration module 410 usesthese to compute various transformations, such as the transformations ofEquations 1, 2, and 3. Registration module 410 then uses thetransformations to determine the baseline model-to-PSC transformationand to generate updates to the model-to-image transformation that isused to overlay model elements on endoscope images. Registration module410 also receives an offset transformation from rendering module 420that are derived from inputs received from an operator 470 on UI device460 as part of the registration process.

Rendering module 420 renders the model in the endoscope image coordinatesystem (e.g., endoscope image coordinate system 250) so that the overlayof the model onto the endoscope image continues to match the underlyinganatomy as established by the registration process. Rendering module 420further renders the model subject to the adjustments made by operator470 during the registration process. Rendering module 420 receives theseadjustments as user interface events received from UI device 460 viainterface module 450.

Overlay module 430 receives the transformed and rendered model fromrendering module 420 and overlays them onto endoscope images from theendoscope to form composite images. Overlay module 430 generates thecomposite images during both the registration process and the subsequenttracking. The model is overlaid onto the endoscope images so that itappears semi-transparent, as for example shown in composite image 350.In some examples, overlay module 430 uses alpha blending and/or the liketo generate the composite images. The composite images are then sent toa display device, such as a stereoscopic viewer, for display to operator470 and/or other personnel.

Optional compression module 440 compresses the composite image and/orthe rendered model in order to reduce storage and/or bandwidth used bythe composite images and/or the rendered model. In some examples,compression algorithms such as JPEG, and/or the like may be used duringthe compression.

Interface module 450 distributes the compressed images to at least UIdevice 460. Interface module 450 may include one or more networkingand/or other APIs for sharing the compressed images via a network.Interface module 450 also receives user interface events from UI device460 and forwards them to rendering module 420 to support theregistration process. Although not shown, interface module 450 and/or aseparate interface module may be used to distribute the composite imagesgenerated by overlay module 430 and that are sent to the display device.

UI device 460 is used, among other things, to display the compositeimages to operator 470 dining the registration process. As operator 470manipulates the model via various interactions with UI device 460, UIdevice 460 forwards these via interface module 450 to rendering module420. These interactions are used to adjust a position and/or orientationof the model during the registration process as is described in furtherdetail below. UI device 460 may also be used to control the appearanceand/or display modes of the composite images during both theregistration process and/or tracking. In some examples, UI device 460may be a touchscreen device, such as an iPad or tablet, a telestrator,and/or the like.

FIG. 5 is a simplified diagram of the method 500 of overlaying imagesaccording to some embodiments. One or more of the processes 510-580 ofmethod 500 may be implemented, at least in part, in the form ofexecutable code stored on non-transient, tangible, machine readablemedia that when run by one or more processors (e.g., the processor 140in control unit 130) may cause the one or more processors to perform oneor more of the processes 510-580. In some embodiments, method 500 may beused to generate composite images, such as composite image 350, whichoverlay a semi-transparent model of anatomy onto images of the anatomycaptured by an endoscope. In some embodiments, method 500 may beimplemented using data flow model 400. In some embodiments, one or moreof processes 510-580 may be performed in an order other than the impliedorder of FIG. 5 and/or may include additional processes as would beunderstood by one of ordinary skill in the art. In some examples,processes 520 and/or 530 may be performed concurrently with process 540.In some examples, process 570 may be performed concurrently withprocesses 550 and/or 560. And although method 500 is described belowwith respect to single images, it is understood that method 500 isadaptable to a stereoscopic vision system where corresponding left andright images may be generated according to the representative processesof method 500.

At a process 510, a model is obtained. In some examples, pre-operativeor intra-operative image data of the anatomy of a patient is obtainedusing a suitable imaging technology such as, CT, MRI, fluoroscopy,thermography, ultrasound, OCT, thermal imaging, impedance imaging, laserimaging, nanotube X-ray imaging, and/or the like. The pre-operative orintra-operative image data may correspond to two-dimensional,three-dimensional, or four-dimensional (including e.g., time based orvelocity based information) images. The image data is then used todetermine the structure of the anatomy including surface features (suchas a parenchyma and/or the like), interior features (such as ducts,calyces, and/or the like), target features (such as a tumor, lesion,and/or the like), and/or the like. In some examples, the model may bepartitioned to identify different structures within the anatomy so thatthey may be selectively omitted from the model and/or rendered using adifferent pattern, false color, and/or the like. In some examples, themodel may be normalized in size and/or maintained in a true-sizecoordinate system. In sonic examples, a centroid of the model may alsobe determined with the centroid being associated with the model as awhole or just to a subset of the structures within the model. In someexamples, the determination of the centroid may depend on the anatomyinvolved, a procedure being performed, operator preference, and/or thelike.

At a process 520, an image is obtained. In some examples, the image maybe obtained using an imaging device, such as an endoscope, and maycorrespond to those portions of the anatomy that are visible to theimaging device. In some examples, the image may be consistent withendoscope image 300. In some examples, the imaging device should bepositioned and/or oriented so as to capture images of the anatomycorresponding to the model obtained during process 510.

At a process 530, the model is overlaid on the image to create acomposite image. In some examples, the model obtained during process 510is initially overlaid onto the image obtained during process 520according to a best-guess as to the relationship between the model andthe image. In some examples, the centroid of the model determined duringprocess 510 is overlaid on the image at a depth from the imaging devicethat roughly corresponds to a current focal length of the imagingdevice. In some examples, this depth may correspond to a locationapproximately 5 to 10 cm in front of the endoscope. In some examples,alpha blending and/or other suitable approach may be used to make themodel semi-transparent so that when it is overlaid on the image, thecontent of the image is still visible despite the presence of the modelwithin the composite image. In some examples, the model may by overlaidvisually such that it includes patterning, false coloring, and/or thelike in order to help the operator distinguish between the content ofthe model and the content of the image.

At a process 540, the model is registered to the image. In someexamples, the operator uses one or more gestures and/or inputs on a UIdevice, such as UI device 190 and/or UI device 460, to manipulate themodel so that the structures in the model match the correspondingportions of the anatomy visible in the image. In some examples, theinputs are used to control a depth (e.g., how far the model is from theanatomy as seen in the image), a position (e.g., a location of the modelrelative to a center of the image), and/or an orientation (e.g., arotation of the model relative to the image). In some examples, variouscontrols such as dials, sliders, and/or the like may be used to obtainthe inputs.

According to some embodiments, various touchscreen and/or telestratoractions may be used to obtain the inputs. As an example, a two-fingerpinch operation may be used to adjust the depth of the model with apinching or closing gesture between the two fingers being used to movethe model further from the endoscope and an opening gesture being usedto move the model closer to the endoscope. Changing the depth of themodel relative to the endoscope also gives the appearance of scaling thesize of the model relative to the image. A two-finger drag operation maybe used to translate the model left and right and/or up and downrelative to the image. A two-finger rotate operation may be used torotate within the plane of the image and thus change a roll orientationof the model relative to the image via a rotation about the direction ofview. A one-finger drag operation may be used to control a yaw of themodel (amount of horizontal drag) and/or a pitch (amount of verticaldrag). Each of the rotations may occur either about the centroid of themodel or about a coordinate system defined by the focal distance of theimaging device, the direction of view, and the view-up orientation ofthe image. A sensitivity of the manipulations to the model based on theamount of one- and/or two-finger movement may be set according to aprocedure being performed, operator preference, and/or the like. Whenthe operator is satisfied that the model is acceptably registered to theimage, the operator may indicate this by activating an appropriate input(e.g., a button) on the UI device and/or some other input deviceassociated with the system performing method 500.

During process 540, each of the manipulations of the model relative tothe image is composited together to generate a model-to-imagetransformation, such as model-to-image transformation 255. Equation 1may then be used to determine a baseline model-to-PSC transformation orequivalent transformation for use during later processes.

At a process 550, operation of a computer-assisted device used toperform the procedure is monitored. This includes determining whetherthere are any changes in the kinematic relationships of thecomputer-assisted device so that the effects of these changes in themodel-to-image transformation may be accounted for.

At a process 560, the model-to-image transformation is updated. Themonitored changes observed during process 550 are used to update thetransformations that describe the kinematics of the computer-assisteddevice using the approach described previously with respect to FIG. 2.Equation 3 may then be used to update the model-to-image transformationso that the overlaid model may be automatically adjusted to track themovements so that the model remains registered to the image.

At a process 570, an image is obtained using a process similar toprocess 520, and at a process 580 the model is overlaid on the imageusing a process similar to process 530 with the overlay occurringaccording to the updated model-to-image transformation determined duringprocess 560. Monitoring of the operation of the computer-assisted devicecontinues by returning to process 550.

As discussed above and further emphasized here, FIG. 5 is merely anexample which should not unduly limit the scope of the claims. One ofordinary skill in the art would recognize many variations, alternatives,and modifications. In some embodiments, method 500 may be adapted toaccount for observed motion in the surgical table and/or of the patientrelative to the table, such as by using table registration and thedetermination of a table-to-PSC transformation as described with respectto FIG. 2. In some embodiments, method 500 may be altered to allowadditional registration processes (e.g., by repeating processes520-540). In some examples, an additional registration process may beinitiated by the operator using a suitable input and/or by detection ofthe system that the model is no longer suitably matched to the image(e.g., by detecting excessive movement in the computer-assisted deviceand/or surgical table).

In some embodiments, one or more other imaging techniques may be usedduring method 500. In some examples, the registering of processes520-450 may be performed with respect to an image obtained using a firstimaging device installed on a first articulated arm of thecomputer-assisted device and the overlaying and composite imagegeneration of processes 570 and 580 may be performed with an imageobtained using a second imaging device installed on a second articulatedarm of the computer-assisted device. In some examples, the imageobtained during process 520 may be obtained using fluorescence imagingand/or multi-spectral imaging and may include images of one or moresubsurface anatomical structures, such as lesions, tumors, vasculature,ducts, calyces, and/or the like. In some examples, the subsurfacefeatures highlighted by the fluorescence imaging and/or multi-spectralimaging may be used to support the registration of process 540.

Some examples of control units, such as control unit 130 may includenon-transient, tangible, machine readable media that include executablecode that when run by one or more processors (e.g., processor 140) maycause the one or more processors to perform the processes of method 500.Some common forms of machine readable media that may include theprocesses of method 500 are, for example, floppy disk, flexible disk,hard disk, magnetic tape, any other magnetic medium, CD-ROM, any otheroptical medium, RAM, PROM, EPROM, FLASH-EPROM, any other memory chip orcartridge, and/or any other medium from which a processor or computer isadapted to read.

Although illustrative embodiments have been shown and described, a widerange of modification, change and substitution is contemplated in theforegoing disclosure and in some instances, some features of theembodiments may be employed without a corresponding use of otherfeatures. One of ordinary skill in the art would recognize manyvariations, alternatives, and modifications. Thus, the scope of theinvention should be limited only by the following claims, and it isappropriate that the claims be construed broadly and in a mannerconsistent with the scope of the embodiments disclosed herein.

1. A method of generating augmented reality images comprises:registering a model of patient anatomy to a first image of the patientanatomy captured using an imaging device to determine a baselinerelationship between the model and the first image; tracking movement ofa computer-assisted device used to manipulate the imaging device;updating the baseline relationship based on the tracked movement; andgenerating a composite image by overlaying the model on a second imageof the patient anatomy according to the updated relationship. 2.(canceled)
 3. The method of claim 1, further comprising rendering themodel so that it is semi-transparent.
 4. (canceled)
 5. The method ofclaim 1, wherein the model is overlaid as a picture within the compositeimage. 6-7. (canceled)
 8. The method of claim 1, wherein registering themodel to the first image comprises: overlaying the model on the firstimage so that a centroid of the model is located along a direction ofview of the imaging device; adjusting the model relative to the firstimage based on one or more inputs received from a user; and generating amodel to image transformation based on the adjustments to the model, themodel to image transformation capturing the baseline relationshipbetween the model and the first image. 9-17. (canceled)
 18. The methodof claim 1, wherein: the imaging device supports fluorescence imaging ormulti-spectral imaging; and the first image includes one or moresubsurface anatomical structures highlighted by the fluorescence imagingor the multi-spectral imaging that are usable for the registering. 19.(canceled)
 20. A computer-assisted medical device, the devicecomprising: an articulated arm; an imaging device mounted on thearticulated arm; and a processor coupled to the articulated arm and theimaging device, the processor being configured to: register a model ofpatient anatomy to a first image of the patient anatomy captured usingthe imaging device to determine a baseline relationship between themodel and image; track movement of the imaging device; update thebaseline relationship based on the tracked movement; and generate acomposite image by overlaying the model on a second image of the patientanatomy according to the updated relationship.
 21. (canceled)
 22. Thedevice of claim 20, processor is further configured to render the modelso that it is semi-transparent.
 23. (canceled)
 24. The device of claim20, wherein the model is overlaid as a picture within the compositeimage. 25-26. (canceled)
 27. The device of claim 20, wherein to registerthe model to the first image, the processor is configured to: overlaythe model on the first image so that a centroid of the model is locatedalong a direction of view of the imaging device; adjust the modelrelative to the first image based on one or more inputs received from auser; and generate a model to image transformation based on theadjustments to the model, the model to image transformation capturingthe baseline relationship between the model and the first image.
 28. Thedevice of claim 27, wherein to register the model to the first image theprocessor is further configured to comprises generate a baselinetransformation between the model and the computer-assisted medicaldevice based on the model to image transformation.
 29. The device ofclaim 28, wherein to updating the baseline relationship is further basedon the baseline transformation between the model and thecomputer-assisted medical device.
 30. The device of claim 27, whereinthe one or more inputs adjust a relative position, a relativeorientation, or both between the model and the second image.
 31. Thedevice of claim 27, wherein the one or more inputs are based on one ortwo finger gestures by the user on a user interface device. 32-33.(canceled)
 34. The device of claim 20, wherein the model comprisesanatomical features not visible in the first image. 35-36. (canceled)37. The device of claim 20, wherein: the imaging device supportsfluorescence imaging or multi-spectral imaging; and the first imageincludes one or more subsurface anatomical structures highlighted by thefluorescence imaging or the multi-spectral imaging that are usable forthe registering.
 38. (canceled)
 39. A non-transitory computer-readablemedium having stored thereon a plurality of machine-readableinstructions which when executed by one or more processors associatedwith a computer-assisted device are adapted to cause the one or moreprocessors to perform a method comprising: registering a model ofpatient anatomy to a first image of the patient anatomy captured usingan imaging device to determine a baseline relationship between the modeland the first image; tracking movement of a computer-assisted deviceused to manipulate the imaging device; updating the baselinerelationship based on the tracked movement; and generating a compositeimage by overlaying the model on a second image of the patient anatomyaccording to the updated relationship.
 40. (canceled)
 41. Thenon-transitory computer-readable medium of claim 39, wherein the methodfurther comprises rendering the model so that it is semi-transparent.42. (canceled)
 43. The non-transitory computer-readable medium of claim39, wherein the model is overlaid as a picture within the compositeimage. 44-45. (canceled)
 46. The non-transitory computer-readable mediumof claim 39, wherein registering the model to the first image comprises:overlaying the model on the first image so that a centroid of the modelis located along a direction of view of the imaging device; adjustingthe model relative to the first image based on one or more inputsreceived from a user; and generating a model to image transformationbased on the adjustments to the model, the model to image transformationcapturing the baseline relationship between the model and the firstimage. 47-55. (canceled)
 56. The non-transitory computer-readable mediumof claim 39, wherein: the imaging device supports fluorescence imagingor multi-spectral imaging; and the first image includes one or moresubsurface anatomical structures highlighted by the fluorescence imagingor the multi-spectral imaging that are usable for the registering. 57.(canceled)